Annular area preserving homeomorphisms which admit no 
interior compact invariant sets 



Shigenori Matsumoto 

Abstract. For any irrational number a, there exists an ergodic area preserv- 
ing homeomorphism of the closed annulus which is isotopic to the identitity, 
admits no compact invariant set contained in the interior of the annulus, and 
has the rotation number a. 



1. Introduction 

Let A = S 1 x [— 1, 1] be the closed annulus equipped with the standard area. 
Denote by G r (r = 0, 1, • • • , oo) the group of the area preserving C diffeomorphisms 
of A which are isotopic to the identity, and by d±A = S 1 x {±1} the upper or lower 
boundary curves of the annulus. In |Hel] . Michael Hermann announced that if 
/ € G°° has no periodic points and if the rotation number of / restricted to <9_A 
is a non Liouville number, then / is C°° conjugate to a rigid rotation near 9_A. 
According to [FSj . in fact he seems to have shown the following. 

If the rotation number of f\d_& is non Liouville, then there is a family of 
invariant smooth circles accumulating to cLA. 

B. Fayad and M. Saprykina )FSj then showed the optimality of this result by 
proving that for any Liouville number a, there is / € G°° such that / is weakly 
mixing and that the rotation number of the two boundary curves is a. 

In this paper we consider a somewhat different topological counter part of the 
Herman phenomenon. A map in G r is called a NICIS map if it admits no compact 
invariant set contained in the interior of A. 

Let A = W. x [—1, 1] be the universal cover of A and p : A — >• M be the projection 
onto the first factor. Fix once and for all a lift / : A — > A of / e G r . Then the map 
P ° f ~ p : A R is invariant by the covering transformation group and defines a 
map on A, which we shall denote by the same letter p o f — p. For a NICIS map 
f £ G , there is an irrational number a such that for any /-invariant probability 
measure /i on A, we have 

(p o / - p, fj) = a. 

See Appendix A. The number a is called the rotation number of / and is denoted 
by p{f). The main result of the present paper is the following. 
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Theorem 1. (1) If a is a Liouville number, then there is a NICIS C°° - diffeomor- 
phism f (and its lift f) such that p(f) = a. 

(2) For any irrational number a, there is a NICIS homeomorphism f such that 

P(f) = <*■ 

Recall that an irrational number a is called a Liouville number if for any r > 
and e > 0, there is a rational number p/q, (p, q) — 1, such that \a — p/q\ < e/q T . 

The proof of (1), given in Sect. 2, is based on the fast approximation by con- 
jugacy method developed in [FSj . But since the NICIS diffeomorphism is much 
simpler to handle than the weakly mixing diffeomorphism treated in [FSj . our ar- 
gument becomes vastly easier than in |FSj . However, as will be apparent in Sect. 
2, it looks hopeless to prove (2) by this method, and we show it in Sect. 3 by con- 
structing a conjagacy with a Anzai skew product F : S 1 x R — > S 1 x M over the 
rotation R a : S 1 — > S 1 by a. The obtained homeomorphism is not weakly mixing, 
although it is ergodic. In Appendix A we show some fundamental properties of 
NICIS maps and in Appendix B we prepare necessary prerequisites about Anzai 
skew products. 

2. Fast approximation by conjugation method 

Define the C r -distance d r on G r by 

d r (/,ff)=max{||/-. g || r ,||/- 1 -.g- 1 || r }, 

where |j • |j r denotes the C r -norm. Let us denote the standard 5 1 -action on A by 
St (t G S 1 ): 

St{x,y) = (x + t,y). 
For any n G N, choose an increasing sequence of positive numbers a n tending 
to 1. Let 

U„ = S 1 X ([-l,-On)U (On,l]) 

and 

G r n = {f g G r | Of(x) nU n ^Q),Vxe A}, 
where Of(x) denotes the /-orbit of x. Notice that G r n is C°-open in G r and that 
/ G G r is NICIS if and only if / G (~] n G r n . 

For the moment let a be an arbitrary irrational number. We shall construct a 
diffeomorphism h n G G°° and a rational number a n — p n /q n (n G N) such that 

(A) h n o S an = S an o h n . 
Denoting 

H n = hi o hi o • • ■ o h n , fn = H n o S an+1 o H^ 1 , 
we will also require the following. 

(B) The map f n converges to some f G G°° in the C°° -topology and a n — > a. 

(C) For any n, the C° '-closure of the set {f m \m > n} is contained in G® n . 

Then the limit / is a NICIS C°°-diffeomorphism and by the continuity of the 
rotation number we have p(f) = a for some lift /. The condition (A) plays a key 
role in getting (B) and (C), since it implies the equalities 

fn+1 - fn = H n+ i O S an+2 O H n+1 - H n +l ° S a „ +1 ° H n +H 
f n +l ~ fn 1 = H n + 1 o 5_ Q „ + 2 o H n+1 - H n+1 o S^ an + 1 o H n ^, 

which enable us to estimate d r (f n +i, fn) for example by Lemma \2 . 1 1 below. 
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Assume we already defined hi, • • • , h n -i and ai, • • • ,a n — Pn/qn- The con- 
struction of h n is carried out in the following way. We identify the cyclic g„-covering 
space of A with A in a standard way. First we construct k n £ G°° which keeps the 
vertical line Iq = x [—1, 1] invariant, and let h n be the lift of k n which keeps Iq 
invariant. Then clearly h n commutes with Si/ qn , and hence with S an . 

The construction of k n goes as follows. Fix real numbers b n and c„ so that 
a-a < b n < c n < a n+ \. Choose k n £ G°° such that k n is the identity on the 
neighbourhood E/ n +i and on the vertical line Iq and that k n maps the horizontal 
curve S 1 x to the graph of the function y — c n sin27ra;. The Moser lemma (|M ) 
shows the existence of such an area preserving diffeomorphism. 

Notice that the orbit foliation of the ^-action St is mapped by k n to a foliation 
all of whose leaves intersect S 1 x ([— 1,— c„] U [c n , 1]). Clearly this property is 
inherited to h n and also to H n = hi o • • • h n -i ° h n since hi o ■ ■ ■ h n -i is the identity 
on U n . That is, any orbit of the ^-action H n o St ° H^ 1 (t £ S 1 ) intersects 
S 1 x([-l,-c n ]U[c n ,l]). 

Next choose a rational number a„+i = Pn+i/ln+i which approximates the 
given a so as to satisfy the following conditions (2.1), (2.2) and (2.3) below. 

(2.1) The number q n +i is big enough so that any orbit of the diffeomorphism H n o 
Si/q n+1 ° H^ 1 intersects S 1 x ([— 1, — b n ] U [b n , 1]). We also assume 

\a - a n+ i\ < \a — a n \ 

for any n. 

The same intersection property is enjoyed also by the diffeomorphism /„ = 
H n ° S an+1 o H~ , since the both diffeomorphims, periodic of order q n +i, have the 
same orbit structure. 

Furthermore if g £ G° satisfies 

h 3 - fnh < b n ~a n (1 < j < q n+ i), 
then g £ G° , i. e. any orbit of g intersects U n . Let 

. r b n ~a n b n -i — a n -i bi — a\ 



If we choose diffcomorphisms /„ = H n o S an+1 o H n 1 in such a way that 

\\fi+i-ti\\o<e n (l<j<q n+ i) 

for any n, then the sequence {f n } satisfies (C), and if further /„ converges to some 
/ in G°, then / belongs to n„G°, that is, / is a NICIS map. 
Denoting the Lipschitz constant by L(-), we have 

ll/n+l-/nllo = ll^n+l°^ ctn+2 o^ r H 1 i--ffr l +lo5' :)ctn + 1 oi/ J 7 4 L 1 || < L(H n+1 )j\a n+2 -a n+ l\. 

Since L(h n ) < L(k n )q n , 1 < j < q n+x and 

\a n +2 ~ a n+ i\ < 2\a - a„ +1 | 

by (2.1), we get 

ll/n+i - fnh < C n qi ■ ••g„^ +1 2|a - a„+i|, 
where G„ = L(ki) ■ ■ ■ L(k n ). 

Thus the following condition (2.2) assures \\fn+i — fih < £ n- 
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(2.2) We have 

I Pn+l, £n 1 

I" I < ^ 2 • 

Qn+1 &^n1l " 'In Qn+l 

Finally to guarantee the convergence of f n , we need the following lemma. The 
statement (1) is just Lemma 5.6 of |FSj . and (2) is easy to establish. 

Lemma 2.1. (1) For r > 0, a, (3 £ R and for H g G r , u>e /iawe 

d r {HoS a oH-\HoS, ) oH- 1 ) < C 2 (r)\\H\\ r r +[\a-f3\, 

where C^r) is a constant depending only on r. 
(2) We have 

\\K\\r < C 3 (n,r)q^, 

where C^{n,r) is a constant depending only on n and r. □ 
The lemma implies that 

dr(/„+l,/«) < C 4 (n,r)q[ r+1)2 ■ ■ ■ q£+»' q£+»* \a - a n+1 \, 

where Ci(n, r) is a constant depending only on n and r. 

Therefore the following (2.3) garantees that d n (f n +i,f n ) < 1/2™ and that /„ 
converges in G°° . 

(2.3) We have 

, Pn+l , , 1 1 

a < 



ftH-l- 2™C 4 (n ; n)^ +1)2 ---^ +1)2 ' 

Now if a is a Liouville number, then it is possible to choose p n /q n successively 
so as to satisfy (2.1), (2.2) and (2.3) above. This shows (1) of Theorem [TJ 

Notice that if we are working in G°, the condition for C°-convergence becomes; 

(2.3') 

\a < 



q n+ i ~ 2™C 4 (n, 0)qi •■■(?„ 9n+i ' 
and any irrational number a admits a sequence {p n /qn} satisfying (2.1) and (2.3'). 
On the other hand the condition (2.2) cannot be relaxed, and it remains still nec- 
essary to require the well approximability of a. Recall an irrational number a is 
called well approximable if for any e > 0, there is a rational number p/q such that 
\a-p/q\ < e/q 2 . 

3. Anzai skew products 

The purpose of this section is to give a proof of Theorem Q] (2). Denote by 
R a : S 1 — > S 1 the rotation by a, and let tp be a real valued continuous function on 
S 1 . Define a homeomorphism F a ^ of the annulus S 1 x R by 

F a , v (x,y) = {R a (x),y + <p(x)). 

The homeomorphism F a-ip is called a Anzai skew product over R a . Throughout this 
section we assume 

(3.1) The rotation number a is irrational. 

(3.2) The function ip satisfies J gl (p(x)dx = 0. 

(3.3) The function ip is nonintegrable, i. e. the functional equation h o R a — h = (p 
has no continuous solution h : S 1 — > K. 
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(3.4) The function ip> is absolutely continuous. 

Notice that if condition (3.2) is not fulfilled, then the unique ergodicity of R a 
implies that all the orbits of F a ip are discrete in S 1 x R and there is no F av - 
invariant probability measure on S 1 x R. Also if there is a solution h in (3.3), then 
the graph of h as well as its vertical translates will be an invariant curve. On the 
other hand conditions (3.3) implies that there are no compact invariant set of F a ^ v 
(Theorem 14.11, [GH] i 

For a point (x, y) E S 1 x R, denote by A(x, y) the limit set of (x, y) by the 
homcomorphism F a ^ v , i. e. the union of the a-limit set and the w-limit set: 

A(x,y) = a(x,y) U w(x,y). 

Following H. Poincare, define 

Lx = {yeR| (x,y) G A(x,0)}. 

Since F a ^ v commutes with the vertical translations, L x forms a closed subsemi- 
group of R, and we have 

A(x,y) fl(ixl) = (x,y + h x ). 

Define 

D = {x G S 1 | L x = R}, 

and notice that x G 13 if and only if the orbit of (x, y) is dense in S 1 x R for some 
(any) y € R. Also define 

P = {x G S 1 | h x = [0, oo)}, N = {x G S 11 | L K = (-oo, 0]}. 

The following theorem plays a crucial role in what follows. It was proven in [K] 
for C^-function ip in the situation where the A-limit set is replaced by the w-limit 
set. But this is not enough for our purpose. We shall include the proof in Appendix 
B. 

Theorem 3.1. Assume that F a ^ satisfies (3.1) ~ (3.4). Then all the three subsets 
D, P and N are nonempty and they constitute a partition of S 1 into Borel sets. 

Here is an outline of the proof of Theorem [T] (2). Given any irrational number 
a, we shall construct a function ip : S 1 — > R satisfying (3.2) ~ (3.4). Moreover ip is 
to satisfy (3.5) and (3.6) below. 

(3.5) There exists a function h G L 1 (S l ,dx) such that ho R a — h = ip, where dx 
denotes the Lebesgue measure on S 1 . 

(3.6) We have tp{—x — a) = ip(x) for any x G S . 

. 

Wc identify S x R with the interior A of the closed annulus A in a standard way 
by a diffeomorphism which maps a fiber x x R onto a fiber x X (— 1, 1). Clearly the 
map F a ^ v extends to a homeomorphism of the closed annulus A in such a way that 
the restriction to each boundary component is the rotation by a. Using a solution 
h of (3.5), let us define a measurable map T(h) : S 1 — > A by T(h)(x) = (x, h(x)). 
Then we get a probability measure (i — r(ft,)*o?x on A. Condition (3.5) implies that 
[i is invariant by F a , v . 

Clearly fi is nonatomic, and fi(dA) = 0. Furthermore once we show that \i is of 
full support, we can find a homeomorphism g of A, identity on the boundary, such 
that g*[i is the standard area of A f |OU] 1. Then the conjugate g o F a>v o g _1 is a 
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NICIS homeomorphism with rotation number equal to a. Also it is ergodic since 
it is metrically isomorphic to the rotation R a on S 1 . 

Given a, the function ip is constructed as follows. For x £ R, ||x|| denotes the 
distance in S 1 = R/Z of the class of x to 0. Choose a sequence {q n }^Li of positive 
integers satisfying 

(3.7) ||g„a|| < l/q n , (3.8) q n+1 > 10q n , 

and define 

N 1 

<p N ( X ) = V _(( e 2 ™3" a - l) e ^i<!nx _ / e -2™ 9 „ Q _ x y-2iriq n xy 

ni 

n=l 

We have by (3.7) 

| e ±*rffc« _ 1| < 27r|| ? „a|| < — . 

q n 

This, together with the assumption (3.8) shows that ipN converges uniformly to a 
continuous function ip. To show that <p is absolutely continuous, we have; 

W'{x)\dx < ( / \p'{x)\ 2 dx) 1/2 
Js 1 

< (2 ]T ^|e 2 " 9 " Q - l| 2 (2^„) 2 ) 1 /2 < 4V27T 2 E ^ < °°- 

n— 1 n 

Since ip' G L 1 (S 1 ,dx), the function ^ is absolutely continuous. 

The Fourier series of the solution h of the equation h o _R a — ft = (p is given by 

oo ^ oo 1 

h(x) = V — (e 2 ™ 9 "* - e- 2 ^"") = V — sin27rg n a;. 
^ — ' ni ^— ' 2n 

n— 1 n— 1 

Direct computation shows that h € L 2 (S 1 ,dx) C L 1 (5' 1 , dec). On the other hand ft. 
is not continuous. To see this, recall that the continuity of h is equivalent to the 
uniform convergence of the associated Cesaro sum (Theorem 73 of |HR ). Let 

C n = {x e S 1 | sin27rg n a; > 1/2}. 

Then by (3.8) a connected component of C n contains a connected component of 
C„+i, and thus one can choose a point xq from the intersection of all the C n 's. 
Then clearly the Cesaro sum diverges at xq. 

The proof that the function ip satisfies the conditions (3.2) ~ (3.5) is now 
complete, while the condition (3.6) can be verified by a direct computation. 

Finally let us show that the measure /i has full support. Notice that condition 
(3.6) implies that 

J o F a , v o J = F~l, 
where J is the involution of S 1 x R defined by 

J fay) = (~x,-y))- 

Since our definition of is invariant by time reversion, this shows that —P = 
N on S 1 . Therefore the measures of P and N must be the same. But since R a is 
ergodic, this implies that D is of full measure in S 1 . 

Suppose for contradiction that the support S of the measure fj, is not the whole 
S 1 x R. Then for any x € D and t/eR, the point (x, y) cannot belong to S, for if 
it did, then we would have S = S 1 x R since the orbit of {x, y) is dense in S 1 x R. 
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Therefore for the projection q : S 1 x R — > S 1 , we have q*fi(D) = 0. But is just 
the Lebesgue measure dx, contradicting the fact that D is of full measure. This 
completes the proof of Theorem [T] (2). 

4. Appendix A: Properties of NICIS homeomorphisms 

Let / be an area preserving homeomorphism of the annulus A, isotopic to the 
identity. Fix once and for all a lift / of / to the universal covering space A of A. As 
before the function p o / — p, defined on A and invariant by the deck transformation 
group, is considered to be a function defined on A. 

Denote by 'Pf(A) the space of the probability measures on A invariant by /. 
Define 

ft(/) = {<P°/-p,M>lA«€P/(A)} 

and call it the rotation set of f. The rotation set is a compact interval (maybe one 
point) of M. 

Proposition 4.1. If f is a NICIS homeomorphism, then the rotation set IZ(f) 
consists of a single irrational number a. 

Proof. It is shown in |Flj that if 71(f) is not a singleton, then there is a 

o 

periodic orbit of / in the interior A of A, contradicting the condition of a NICIS 
homeomorphism. Assume that the rotation set 7t(f) consists of a rational number 
a. In the ergodic decomposition of the Lebesgue measure, there must exist an 

o 

ergodic component v whose support has nonempty intersection with A- Since 

(po / - p,v) = a, 

o 

again there must be a periodic orbit in A, by Proposition 2.3 of [FH2 . □ 

Next we shall show some topological properties of the orbits of NICIS homeo- 
morphisms. 

Proposition 4.2. A NICIS homeomorphism f admit dense orbits. 

o 

PROOF. We shall show that any /-invariant open subset U C A is dense in 

o o 

A- By the Baire property of A, this will imply the existence of dense orbits. In 

o 

fact, choose a countable open basis Uj of A- The set Of(Uj) of points whose orbits 

o 

intersect Uj will then be an open dense subset of A, and a point in the nonempty 
intersection C\jOf(Uj) will have a dense orbit. 

o 

Suppose U is an /-invariant open subset of A- Assume U admits an inessential 

o 

connected component V, i. e. a component for which the inclusion V — > A induces 
the trivial map on the fundamental group. Then V is left invariant by some power 
of /, say f m , since / is area preserving. Let V be the union of all the closed discs 

o 

in A whose boundary is contained in V. Then f m leaves the open disc V invariant, 
and admits a fixed point in V by the Brouwer plane fixed point theorem, since all 
the points of V are nonwandering w. r. t. f m . This contradiction shows that any 
component of U must be essential. 
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o 

Assume for contradiction that U is not dense in A. By the same argument 
applied to the interior of the complement of U, we obtain two disjoint essential /- 

o 

invariant open subset of A. This implies the existence of interior compact invariant 
set. A contradiction. □ 

Next proposition due to |LY) and |F2] shows that NICIS homeomorphisms 
admit nondense orbits as well. Let us denote d±A = S 1 x {±1}. 

Proposition 4.3. Given any neighbourhood U of d+A (or of d-A), a NICIS home- 

o 

omorphism f admits an orbit contained in U PI A- 

5. Appendix B: Proof of Theorem 3.1 

Throughout this section we assume that the skew product F aip satisfies the 
conditions (3.1) ~ (3.4). 

Computation shows that for any integer n > 0, 

n-i 

K,^,y) = (K(x),y + fn(x)), where cp n (x) = ^ viKi*))- 

i=0 

A positive integer q is called a closest return time for a if ||ja|| > \\qot\\ for any 
integer j such that < j < q, where || • || denotes the distance to of the projected 
image in M/Z. The following theorem can be found in Sect. 1.2 of GLL . 

Theorem 5.1. (Improved Denjoy-Koksma theorem) If tp satisfies (3.2) and (3.4), 
and q n is a sequence of closest return times for the irrational number a, then we 
have 

sup|^ g J -> 0. 

It follows at once that for any x £ S , the closed subsemigroup L K is nonempty. 
We also need the following lemma, the proof of which is left to the reader. 

Lemma 5.2. For integers < n\ < n-i < n% assume that is the smallest positive 
integer j such that ja lies in the smaller interval bounded by n\a and n^a. Then 
nz — Ha is a closest return time for a. □ 

The following proposition plays a key role in the proof of Theorem 3.1. 

Proposition 5.3. For any igS 1 and for any < a < b, we have 

L x n ([-&, -a] U [a, b]) + 0. 

Proof. It is no loss of generality to assume that the point x in Proposition 
15.31 is 0. For contradiction assume that the limit set A(0,0) = a(0, 0) U w(0,0) is 
disjoint from the set x ([—&,— a] U [a, b}). Thus one can choose a small positive 
number e such that the union of two rectangles 

Y' = {(x,y) | \x\<e, a < \y\ < b} 

is disjoint from the orbit of (0, 0). 

Claim. For even smaller 5 > 0, the orbit of (0,0) is disjoint from {\x\ < 

S,\y\>a}. 

The claim clearly implies that the orbit of (0, 0) is bounded, which contradicts 
the nonintegrability condition (3.3) by Theorem 14.11 of |GH) . 
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Let us start the proof of the claim. We are going to show this only for the 
positive orbit, the other case being similar. 

Choose positive integers n_ and n+ such that 

(5.1) — e < n_a < < n+a < e, 

(5.2) n± is a closest return time for a, and 

(5.3) if q is a closest return time for a such that q > min{n_,n + }, then 

sup|(y9 9 | < b — a. 

Put 

Y = {(x, y) | n_a < x < n+a, a < \y\ < b}. 

Let us show that whenever a positive iterate of (0,0) by F a ,y> lies in the strip 
(n_a, n+a) x K, it is contained in (n_a, n+a) x (—a, a), that is, n^a < na < n+a 
implies that |</?n(0)| < a. Assume to fix the idea that na € (0,n+a). Define a 
sequence nx,U2,- • • ,nu — n as follows. First of all let n% = n+, and let ni be the 
smallest positive integer j such that ja lies in the interval I\ = (0, n+a). If n% = n, 
the definition is over. If not, n^a divides the interval I\ into two subintervals, one 
of which, say I2, contains na. Let 713 be the smallest positive integer j such that ja 
lies in the interval li- Proceeding in this way, we end up with some n^ eventually 
matching the given n. 

Now by Lemma l5.2l all the integers n\, n% — tvl, •• • , n^ — nk-i are closest return 
times for a. Moreover since \\(rij ~ nj-\)a\ < \\n+a\\, we have nj — nj—i > n+ 
(2 < j < k) and (5.3) implies that sup|<^ n -_ n ._j | < 6 — a as well as sup|<yj ni | < b — a. 

To begin with, we have \<p ni (0)| < b — a < b. But since the point -F™^(0, 0) = 
(R^^^ifimiO)) does not lie in Y, we have in fact |<£> rai (0)| < a. Furthermore we 
obtain 

1^(0)1 = |^ 2 -„ 1 ( J RS 1 (0)) + ^„ 1 (0)| < |y„ 2 -„ 1 (fl2 1 (0))| + 1^(0)1 <(b-a)+a. 

Again since -^^(0,0) does not lie in Y, we actually have |Vn 2 (0)| < a. Proceeding 
this way one gets |vn(0)| < a. 

The proof of the claim and of Proposition 15 . 31 is now complete. □ 

Let us finish the proof of Theorem 13. II Proposition 15 .31 shows that for any x £ 
S 1 , the set is unbounded. If the closed subsemigroup contains both positive 
and negative numbers, then ~h x = M or L T — Z ■ c for some c > 0. Proposition 
15.31 shows that h x = M in this case. Also the same proposition shows that if h x 
consists of nonnegative numbers (resp. nonpositive numbers), then = [0, 00) 
(resp. La, = (—00, 0]). This shows that S 1 is the union of D, P and N. 

What is left is to show that each of the sets D, P and N is nonempty. The 
existence of dense orbits of F a tp is shown in |Hed] under the assumption (3.1) ~ 
(3.3), which implies that D is nonempty. In fact the essential point is to show 
that any point of S 1 x M is nonwandering, the rest being exactly the same as the 
proof of Proposition 14.21 Also in [B] the existence of the F Q V -orbits which are 
bounded below (or above) is shown under the same assumption. This, together 
with Proposition 15.31 shows that P and N are nonempty. 
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